-
Notifications
You must be signed in to change notification settings - Fork 802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(arq): incidents processing #2558
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
WalkthroughThe changes in this pull request involve modifications to two functions within the Changes
Possibly related issues
Poem
Warning Rate limit exceeded@dosubot[bot] has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 15 minutes and 36 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (4)
keep/api/tasks/process_incident_task.py (3)
35-38
: LGTM! Consider minor improvements to context extractionThe addition of job context information is valuable for tracking processing attempts. The implementation is correct, but we can make it slightly more concise.
if ctx and isinstance(ctx, dict): extra["job_try"] = ctx.get("job_try", 0) - extra["job_id"] = ctx.get("job_id", None) + extra["job_id"] = ctx.get("job_id")🧰 Tools
🪛 Ruff
37-37: Use
ctx.get("job_id")
instead ofctx.get("job_id", None)
Replace
ctx.get("job_id", None)
withctx.get("job_id")
(SIM910)
Line range hint
156-158
: Consider implementing exponential backoff for retriesThe current linear retry delay (
job_try * TIMES_TO_RETRY_JOB
) might not be optimal for handling transient failures. Consider implementing exponential backoff to provide shorter initial retries while still allowing longer delays for persistent issues.# Retrying only if context is present (running the job in arq worker) if bool(ctx): - raise Retry(defer=ctx["job_try"] * TIMES_TO_RETRY_JOB) + # Exponential backoff with base of 2 minutes + raise Retry(defer=2 ** ctx["job_try"])🧰 Tools
🪛 Ruff
37-37: Use
ctx.get("job_id")
instead ofctx.get("job_id", None)
Replace
ctx.get("job_id", None)
withctx.get("job_id")
(SIM910)
Line range hint
101-104
: Consider more specific exception handling for alerts processingThe current broad exception handling for alerts processing might mask specific issues. Consider:
- Catching specific exceptions that you expect might occur
- Adding a flag or status to the incident to indicate incomplete alert processing
- Including alert processing status in the pusher notification
Example implementation:
try: # ... alerts processing code ... except (ValueError, KeyError) as e: logger.exception("Error processing alert data: %s", str(e), extra=extra) except ConnectionError as e: logger.exception("Network error processing alerts: %s", str(e), extra=extra) incident_from_db.alert_processing_status = "incomplete" except Exception as e: logger.exception("Unexpected error adding incident alerts: %s", str(e), extra=extra) incident_from_db.alert_processing_status = "failed"🧰 Tools
🪛 Ruff
37-37: Use
ctx.get("job_id")
instead ofctx.get("job_id", None)
Replace
ctx.get("job_id", None)
withctx.get("job_id")
(SIM910)
keep/api/routes/incidents.py (1)
Line range hint
1012-1034
: Fix response model inconsistencyThe endpoint decorator specifies
response_model=List[AlertDto]
, but the function returns aResponse(status_code=202)
. This mismatch could cause issues with OpenAPI documentation and client expectations.Apply this fix:
@router.post( "/{incident_id}/alerts", description="Add alerts to incident", status_code=202, - response_model=List[AlertDto], ) async def add_alerts_to_incident( incident_id: UUID, alert_ids: List[UUID], is_created_by_ai: bool = False, authenticated_entity: AuthenticatedEntity = Depends( IdentityManagerFactory.get_auth_verifier(["write:incident"]) ), pusher_client: Pusher | None = Depends(get_pusher_client), session: Session = Depends(get_session), ): tenant_id = authenticated_entity.tenant_id incident_bl = IncidentBl(tenant_id, session, pusher_client) await incident_bl.add_alerts_to_incident(incident_id, alert_ids) return Response(status_code=202)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
keep/api/routes/incidents.py
(1 hunks)keep/api/tasks/process_incident_task.py
(1 hunks)
🧰 Additional context used
🪛 Ruff
keep/api/tasks/process_incident_task.py
37-37: Use ctx.get("job_id")
instead of ctx.get("job_id", None)
Replace ctx.get("job_id", None)
with ctx.get("job_id")
(SIM910)
🔇 Additional comments (1)
keep/api/routes/incidents.py (1)
552-552
: LGTM! Queue specification for incident processing
The addition of _queue_name=KEEP_ARQ_QUEUE_BASIC
properly directs incident processing jobs to a specific queue, improving job routing control.
Summary by CodeRabbit
New Features
Bug Fixes